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Abstract 

This paper describes the. development of a virtual-machine 
monitor (VMM) security kernel for the VAX architecture. 
The paper particularly focuses on how the system's hard- 
ware, microcode, and software arc aimed at meeting A 1- 
levcl security requirements while maintaining the standard 
interfaces and applications of the VMS and ULTRIX- 32 op- 
erating systems. The VAX security kernel supports multiple 
concurrent virtual machines on a single VA X system, provid- 
ing isolation and controlled sharing of sensitive data. Rigor- 
ous engineering standards were applied during development 
to comply with the assurance requirements for verification 
and. configuration management. The VAX security kernel 
has been developed with a heavy emphasis on performance 
and on system management tools. The kernel performs suf- 
ficiently well that all of its development is now carried out 
in virtual machines running on the kernel itself, rather than 
in a conventional time sharing system. 

1 Introduction 

The VAX security kernel project is a research effort to deter- 
mine what is required to hnild a production-quality security 
kernel, capable of receiving an Al rating from the National 
Computer Security Center. A production ■ quality security • 
kernel is very different from the many research -quality secu- 
rity kernels thai have been built in the past, and this research 
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effort has been primarily aimed at identifying the differences 
and their cost in development effort and in kernel complexity. 

This paper describes how the VAX security kernel meets 
its live major goals: 

• Meet all Al security requirements. 

• Rnn on commercial hardware without special modifica- 
tions other than microcode changes for virlualiaation. 

• Provide software compatibility for applications written 
for both the VMS and ULTRIX 32 operating systems. 

• Provide an acceptable level of performance. 

• Meet the requirements of a commercial software 
product. 

The VAX security kernel is a research effort. Digital. 
Equipment Corporation makes no commitment to offer it 
as a product. 

2 Kernel Overview 

The VAX security kurnol is a virtual-machine monitor 
that runs on the VAX 8530. 8550, 8700, 8800, am) 88 It) 
processors.' Jl creates isolated virtual VAX processors, each 
of which can run other the VMS or ULTRIX- 32 operat- 
ing system. If desired, virtual machines running each of the 
operating systems can run simultaneously on the same com- 
puter system. 3 The VAX architecture was nol vii lualtzablc, 
and therefore extensions were made to the architecture and 
to the processor microcode to support virtuali&ation. (See 
Section 3.2.) 

Figure I show* n typical VAX security kernel configura- 
tion. While the VAX security kernel is a VMM, it is primar- 
ily a security kernel. ThcrcJorr, certain features tradition- 
ally seen in V'MMs, such as scIf-virlualiy.ation or debugging 
uf one VM from another, have been omitted to reduce kernel 
complexity. 

»Tltc VMM d«c* not run on VAX 8820, 8830, or 8840 processors, 
due to microcode and console differences. 

a At least our virtual machine must always run I lie VMS operating 
system, to carry out cerLain system management functions. 
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Figure 1: VAX VMM Security Kernel Configuration 



The VAX security kernel applies both mandatory and dis- 
cretionary access controls to virtual machines. Each virtual 
machine is assigned an acceas class that consists of a secrecy 
class and an integrity class, similar to those in the VMS 
Security Enhancement Service (VMS SES) |5|. The secrecy 
and integrity classes are based on the Bell and LaPadula 
security |2] and Diba integrity |4) models, respectively. The 
VAX security kernel also supports access control lists (ACLs) 
on all objects, similar to those in the VMS operating sys- 
tem [341. 

The VMM security kernel is not a genera] purpose oper- 
ating system. The principal subjects and objects are virtual 
machines and virluaj disks, rather than conventional pro- 
cesses and files. That is the inherent difference between a 
VMM and a traditional operating system. Processes and 
files are implemented within tbe virtual machines by either 
the VMS or ULTIUX-32 operating systems. 

The VAX security kernel can support large numbers of 
simultaneous users. 5 All software development of the VAX 
security kernel is now carried out on several virtual machines 

»Ex££t numbers depend on ihc precise hardware coofiguralioo. 



miming on the VMM on a VAX 88O0 system. On a typical 
day, about 40 software engineers and managers are logged 
in running a mixed load of text editing, <rompilation, system 
building, and document formatting. The system provides 
adequate interactive response lime and is sufficiently reliable 
to support an engineering group that must meet strict mile- 
stones and schedules. As far as we know, the VAX security 
kernel is the first security kernel to support its own devel- 
opment team. The Mul tics Access Isolation Mechanism |36) 
was developed on Mullics itself, but MuJtics with AIM was 
not a security kernel and only received a D2 rating. 

The VAX security kernel is currently in the Design Anal- 
ysis Phase with the National Computer Security Center 
(NCSC) for an A 1 rating. It is being formally specified in Ina 
Jo and formal proofs are being done on the specifications. 



3 Design Approach 

This section describes several of the design choices in the 
VAX security kernel, including details about the virtual ma- 
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cliitie approach lo security kernels, virlualizing Ihc VAX ar- 
chitecture, subjects and objects, access classes, our layered 
design, and other software engineering issues. 

3.1 Virtual Machine Approach 

The choice lo boild the VAX security kernel as a VMM wh* 
driven by two goals; to maintain compatibility with exist* 
ing software written for the VAX architecture and to keep 
software, development and maintenance cods to a minimum. 

Digital Equipment Corporation began plans to enhance 
the security of the VAX architecture in mid- 1970. 0»ir ini- 
tial effort was the design of security enhancements to the 
VMS operating system, first prototyped in 1980 and avail- 
able today in the base VMS operating system and in the 
VMS Security Enhancement Service (5). 

At the lime of I he initial prototype of the VMS secu- 
rity enhancements |1G], Digital considered a traditional ker- 
nel/emulator security kernel to support VMS applications. 
However, it quickly became clear that the software devel- 
opment costs of a VMS emulator would be comparable to 
the cost of development of the VMS operating system itself. 
Worse still, the emulator would have to track all changes 
made to the VMS ojrcrating system, resulting in ongoing 
costs that would be un acceptably high for the limited market 
for Al-sccurc systems. The kernel/emulator system could 
not replace the existing VMS operating system because its 
performance would not be as good, and it would likely be 
export controlled. Furthermore, the growing demand for 
UNIX-based software would force development of a UNIX 
emulator at still more development cost. 

To resolve these development cost and compatibility prob- 
lems, we chose a VMM security kernel approach. A VMM 
security kernel presents the interface of a computer architect- 
lure that is comparatively simple and not subject to frequent 
change. Thus, the VAX security kernel presents an interface 
of the VAX architecture |2lJ and supports both the VMS 
and UJLTR1X 32 operating systems with relatively few mod- 
ifications. 

The idea of a VMM security kernel is not a new one. Mad- 
nick and Donovan |22] first suggested the merits of VMMs for 
security, and Rhode 1 30 J fini I proposed VMM security ker- 
nels. From 197fi to 1982, Systems Development Corpora lion 
(now a division of the UNISYS Corporation) built a ker- 
nelixcd version o*f IDM's VM/370 virtual-machine monitor, 
called KVM/370 |12). While the design of the VAX secu- 
rity kernel is very different from KVM/370. we have applied 
some of the lessons learned in the KVM/370 project [11]. 
Section 7 compares the VAX security kernel with KVM/370. 
Gasscr 1 10, Section 10.7] provides more detail on some of the 
trade-offs between a VMM security kernel approach and a 
kernel/emulator approach. 

3.2 Virtualizing the VAX 

The requirements for virtualizing a computer architecture 
were specified by Popck and Goldberg |2G]. In essence, they 



require that all sensitive instructions and all references to 
sensitive data structures trap when executed by. unprivileged 
code. A sensitive instruction or data structure is one that 
cither reveals or modifies the privileged slate of the proces- 
sor. 

.1.2.1 Sensitive Instructions 

Unfortunately, the VAX architecture does nut meet Popck 
and Goldberg's requirements. Several instructions, includ- 
ing Move Processor Status Long word (MOVPSL), Probe 
(PK013£x), and Return from Exception or Interrupt (RE1) 
are sensitive, but unprivileged. Furthermore, page table en- 
tries (PTEs) are sensitive data structures that can be read 
and written with unprivileged instructions. 

As a result, we marie a number of extensions to the VAX 
architecture to support visualization. In particular, we 
added a VM bit to the processor status longword (PSL) 
that indicated whether or not the processor was executing 
in a virtual machine. A variety of sensitive instructions 
were changed to trap based tin the setting of the VM bit, 
so that the VMM security kernel could emulate their exe- 
cution. Space does not permit a full discussion of the in- 
struction changes, hut some details arc discussed by Kargcr, 
Mason and Leonard [18]. 

3.2.2 King Compression 

The most significant and security- relevant change lo the 
VAX architecture was to virtuah'ze protection rings. In the 
past, only processors with two protection states (such as 
the IDM 3C0/370 architecture) had been virtualized. Gold- 
berg (13. section 4.3) described the difficulties of virlualizing 
machines with protection rings and therefore more than two 
protection states. He proposed several techniques for map- 
ping ring numbers, some in software and one. with a hardware 
ring relocation register, but he recognized that none of his 
techniques were satisfactory. His software techniques broke 
down because the physical ring number remained visible, and 
his hardware ring relocation technique broke down because 
virlualizing a machine with N rings always required N-f I 
rings. 

Since the VMS operating system uses all four of the pro- 
tection rings of the VAX architecture, it was essential that 
wc develop a new technique for visualization of protection 
ring?. That technique is called ring compression. 

Figure 2 shows how the protection rings of a virtual VAX 
processor arc mapped to the rings of a real VAX proces- 
sor. Virtual user and supervisor modes map to their real 
counterparts, but virtual executive and kernel modes both 
map to real executive mode. The real ring numbers an: con- 
cealed from the virtual machine's operating system (VMOS) 
by three extensions to the VAX architecture: the addition of 
the VM bit to the PSL ( described in Section 3.2 A), the *d- 
rlitinn of a VM processor status lortgword register (VMPSL), 
and the modification of all instructions that could reveal the 
n-al ring number. Those instructions cither trap to the VMM 
security kernel for emulation or obtain their information from 
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Figure 2: Ring Compression 

the VMPSL, which contains the virtual ring number rather 
than the real ring number. Additional details can be found 
in Karger, Mason and Leonard [IS]. 

Ring compression also requires thai the security kernel 
change the memory protection of pages belonging lo virtual 
machines so that their kernel-mode pages become accessible 
from executive mode. This change of memory protection 
could adversely affect security within a given virtual machine 
because the virtual machine's kernel mode is no longer fully 
protected from its executive mode. 

For the two operating systems of interest to the VAX se- 
curity kernel, there is no effective loss of security within the 
virtual machines themselves, although there is a loss of ro- 
bustness against potentially bug-laden executive mode code. 
Fortunately, the VMS operating system grants all programs 
that run in executive mode the right to change mode to ker- 
nel at will and uses the the kernel /executive mode boundary 
only as a reliability mechanism. Furthermore, the ULTRIX- 
32 operating system docs not use executive mode at all. 

Of course, the compression of kernel and executive modes 
in the virtual machines in no way compromises the security 
of the VMM, as the security kernel runs ooly in real ker- 
nel mode, and no virtual machine ever is granted access to 
real kernel mode pages. If there were some other VAX op- 
erating system that actually used all four rings for security 
purposes, it would lose some of its own security, much as 
IBM operating systems lose some of their security when run 
in VM/370. However, no such operating systems exist for 
the VAX architecture. 

3.2.3 I/O Emulation 

Traditional virtual- machine monitors, such as IBM's 
VM/370, have virtualizcd not ooly the CPU, but also the 
1/0 hardware. Visualization of the I/O hardware allows 



the VMOS to run essentially unmodified. Visualization of 
the VAX 1/0 hardware is particularly difficult because its 
I/O devices are programmed by reading and writing control 
and status registers (CS1U) that arc located in a region of 
physical memory called I/O space. This type of I/O origi- 
nated on the PDP-11 scries of computers and caused per- 
formance difficulties in the UCLA PDP-11 virtual-machine 
monitor [27) because the VMM must somehow simulate ev- 
ery instruction that manipulates a CSR. Vahcy [33] proposed 
a complex hardware performance assist, but audi a device 
would add excessive complexity and development cost to the 
VAX security kernel. 

Instead, the VAX security kernel implements a special I/O 
interconnection strategy for virtual machines. The VAX ar- 
chitecture 121] does not specify how I/O is lo be done, and 
different VAX processors have implemented very different 
I/O interfaces. The VAX security kernel I/O interface is 
a specialised kernel call mechanism, optimized for perfor- 
mance, rather than traditional CSR-based I/O. In essence, a 
virtual machine stores 1/O-rclatccl parameters (such as buffer 
addresses, etc.) in specified locations in its I/O space, but no 
I/O taxes place until the virtual machine executes a Move to 
Privileged Register (MTPR) matruclion to a special kernel 
call (KCALL) register. This MTPR traps to security kernel 
software that then performs the I/O. Thus, the number of 
traps to kernel software is dramatically less than would be 
required for CSR emulation. 

This special kernel I/O interface means that special un- 
truBled virtual device drivers had to be written for both the 
VMS and ULTR1X 32 operating systems, but this effort was 
no more than is typically required to support a new VAX 
processor, a small number of engineer-years. 

Because the virtual VAX processors have an I/O interface 
different from that of any existing VAX processors, the VAX 
security kernel docs not fall into any of Goldberg's tradi- 
tional categories of VMMs. Goldberg |13, pp. 22-26) defines 
a Type I VMM as a VMM that runs on a bare machine. He 
defines a Type 11 VMM as a VMM that runs under an ex- 
isting host operating system. Goldberg [13, section 3.3| also 
defines a Hybrid Virtual-Machine Monitor na one in which 
all supervisory -state instructions are simulated, rather than 
just the privileged instructions. The VMM security kernel is 
essentially a cross between a self- visualizing Typo I VMM 
for all non-l/O instructions and a Hybrid Virtual-Machine 
Monitor for I/O instructions. 



3.2.4 S el f-Vi realization 

As we designed the extensions to the VAX architec- 
ture, we ensured that the architecture would permit self- 
visualization. Sclf-virtualization is the ability of a virtual- 
machine monitor to run in one of its own virtual machines 
and recursively create second-level virtual machines. Self- 
virtualization is very useful for developing and debugging 
the virtual- machine monitor itself, but it is of little value to 
actual users. Since self- visualization would have added sig- 
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nihcant complexity to the Trusted Computing lia.se (TOD), 
no software support lime \h*:» done. 4 

3.3 Subjects 

There arc two kinds of subjects in the. VAX security kernel, 
users and virtual machines (VMs). A user communicates 
over the trusted path with a process called a Server. Servers 
are trusted processes, out itnlikr. the trusted processes ui 
olh«r systems such as KSOS-1 I [3j, servers run only within 
the kernel itself. User subject* cannot rim user- written code; 
servers execute only verified code that is part of the TCB. 
The powers of a server are determined by: 

• The user's minimum and maximum access class. (Sec 
Section 3.5.) 

• The terminal's minimum and maximum access class. 

• The user's discretionary access rights. 

• The user's privileges. (Sec Section 3.G.) 

• The privileges exercisable from the terminal. 

A virtual machine is an unlruslcd subject that runs a 
VMOS. A user interacts with the VMOS in whatever fashion 
is normal for that operating system, for example, by logging 
into lliat VMOS and issuing commands. A user may write 
and run code inside a VM and even penetrate the VMOS, 
nil without affecting the security of other visual machines 
or the security kernel itself. At worst, a penetrated virluaJ 
machine could only affect other virtual machines with which 
it shared disk volumes. 

On login to the security kernel, the VMM establishes a 
connection between the user's terminal line and the user's 
Server, called a session. When the user wants to use some 
virtual 1 machine, the user issues the CONNECT command to his 
or her Server, specifying the name of that VM. If the con- 
nection is authorised, the system suspends the user's existing 
session with the Server and establishes a new session between 
the user's terminal line and the requested virtual machine. 
Thus, the Servers and the VMs havn distinct identities and 
distinct security attributes. 

Virtual machines may be run in a single- user mode to pro- 
vide maximum individual accountability. Alternately, they 
can be run in a multi user mode. In such a case, individual 
accountability might be achieved by running a VMOS with 

iThr. software changes needed for self- v,rtuafixalk»n primarily con- 
stat of cH*n*o» to tlx-, virtual device drivers described in Section 3.2.3 
and some ciianges in the emulation of certain sensitive instructions. 
Und« th« prorw-ed Trusted VMM Interpret**... (J), it might even b« 
possible for a «ctf-virtu*lr*cd security kernel to iUelf remain A I rated. 
To achieve tluU goal, the first level VMM would map the second level 
VMM's kernel mode to real executive mode, while the VM» running on 
lop of the second level VMM would Have their supervisor, csccnlive, 
and kemd modes all mapped U. real supervisor mode. Of course, as 
nor continues to recursively sclf virtualixc, one runs out or protects, 
rings at the fourth Icrd VMM, and that VMM would no longer be 
protected from its virtual machines. 



at least n C2 ruling, as suited by the proposed Trusted 
VMM Interpretation |l| of Trusted information Systems. Inc. 

Virtual machines can also be treated as objects because a 
user may request that the TCB provide a connection between 
the user's terminal and some VM. For this operation, the 
user is the subject and the VM is the object. 

3.4 Objects 

The VAX sernTity kernel supports a variety of objects in- 
eluding real devices and volumes and security kernel files. 

One group of objects com prists; the real devices on I he 
system: disk drives, tape drives, printers, terminal lines, and 
single access-class network lines. As these devices can con- 
tain or transmit information, access to them must be con- 
trolled by the TCD. Another object is the primary memory 
that is allocated to each VM when it is activated. 

Disk and tape vol moon are also objects. The contents of 
some disk volumes arc completely under the control of a vir- 
tual machine. They may contain a file system structure or 
just an arbitrary coller.tifm of bits, depending on the method 
used by the VMOS to access the volume. Such volumes arc 
railed exchangeable volumes because they may be exchanged 
with other computer systems running conventional operating 
systems. Other disk volumes contain a VAX security kernel 
file structure and are called VAX security kernel volumes. 
These volumes must not be directly accessed by a VMOS 
or exchanged with other systems, as an entrusted subject 
could subvert the kernel's file system or road information to 
which it was not entitled. The VAX security kernel docs not 
provide trusted tape volumes; all tape volumes are exchange- 
able. m 

VAX security kernel volumes contain VAX security kernel 
files that arc orgamVxd as a Hal file system. VAX security 
kernel files are used for a variety of purposes in the system 
and arc considered objects by the TCB. One use for VAX 
security kernel files is to hold long-term system databases 
such as the audit log and the authorization file. These fibs* 
are considered part of the TCD and, with the exception of the 
audit log, error log and crash dump files, cannot be directly 
referenced by virtual machines. 

Another use of VAX security kernel files is to create vir- 
tual disk volumes, loosely analogous to mini-disks in IBM's 
VM/370 |23, pp. 549-5C3|. Mini-disks allow a physical disk 
to be partitioned, so that one need not dedicate an entire 
physical disk to a small virtual machine that only requires a 
small amount of disk spare. Such vwluaJ disks may contain 
the file structure of some VMOS, such as a VMS file struc- 
ture or an ULTRIX 32 file structure. However, the VMM 
deals with virtual disks only as a whole. The contents of a 
virtual disk arc all part of a single object as far as the VMM 
is concerned. 



3.5 Access Classes 

The VAX security kernel enforces mandatory controls, as re- 
quired of all Al systems. Both secrecy and integrity models 
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are supported, based on ihc work of Bell and LaPaduIa [2] 
and of Diba |4), respectively. To implement mandatory con- 
trols, each kernel subject and kernel object is assigned a 
sensitivity label, called an access class.* An access class 
consists of two components, a secrecy class and an integrity 
class. These components arc each further divided into a levcJ 
and a category set. A secrecy level is a hierarchical classifi- 
cation. The secrecy category set is the set of non-hierarchical 
secrecy categories that represents the sensitivity of the ac- 
cess class. The integrity level and integrity category set arc 
defined analogously. For compatibility with VMS SES |5j, 
the kernel supports 256 secrecy levels, 256 integrity levels, 
64 secrecy categories, and 64 integrity categories. 

Given the complex structure of access classes, two defini- 
tions must be carefully constructed: 

Definition 1 An access class A is equal to an access class 
B if and only if: 

• The secrecy level of A is equal to the secrecy level of B, 

• The secrecy category set of A is equal to the secrecy 
category set of B, 

• The integrity level of A is equal to the integrity level of 
B, and 

• The integrity category set of A is equal to the integrity 
category set of B. 

Definition 2 An access class A dominates an access class 
B if and only if: 

• The secrecy level of A is greater than or equal to the 
secrecy level of B t 

• The secrecy category set of A is an improper superset of 
the secrecy category set of B, 

• The integrity level of A is less than or equal to the 
integrity level of B, and 

• The integrity category set of A is an improper subset of 
the integrity category set of B. 

The secrecy and integrity models define that a subject 
may reference an object depending on the access classes of 
the subject and object and on the type of reference. A sub- 
ject may read from an object only if the subject's access class 
dominates the access class of the object. A subject may write 
to an object only if the object's access class dominates the 
access class of the subject. 0 Thus, for example, a virtual 
machine may mount for read-wrile access an exchangeable 
volume only if the VM's access class is equal to that of the 
volume. However, the VM may mount for read-only access 
any exchangeable volume where the VM's access class dom- 
inates that of the volume. 

5 Some objects, such as terminal lines, may be assigned a range of 
access classes. 

*!n general, write access is even further restricted; a subject may 
write to sn object only if the inibjnct'* and object** acce** dames ure 
equal. This disallows blind writes to an object that cannot be read. 



3.6 Privileges 

System managers, security managers, and operators gain 
their powers by having privileges. The privileges allow great 
flexibility in the assignment of powers and responsibilities, 
including a measure of two-person control and separation of 
duties. Privileges restrict access beyond the protections pro- 
vided by mandatory and discretionary access controls. A 
privileged user cannot see data that would be otherwise in- 
accessible. Only the downgrading privileges allow bypassing 
of access controls, and the rise of those privileges is audited. 

Most privileges can be exercised only through the trusted 
path and are called user privileges. (Sec Tabic 1.) Two 
privileges can be exercised by virtual machines and are called 
virtual-machine privileges. (See Table 2.) 

3.7 Layered Design 

The VAX security kernel bas been implemented following the 
strict levels of abstraction approach originally used by Dijk- 
stra |B] in the THE system. Janson |15) developed the use of 
levels of abstraction in security kernel design as a means of 
reducing complexity and providing precise and understand- 
able specifications. Each layer of the design implements some 
abstraction in part by making calls on lower layers. In no 
case docs a lower layer invoke or depend npon higher layer 
abstractions. By making lower layers unaware of higher ab- 
stractions, wc reduce the total number of interactions in the 
system and thereby reduce the overall complexity. Further- 
more, each layer can be tested in isolation from all higher 
layers, allowing debugging to proceed in an orderly fashion, 
rather than haphazardly throughout the system. This type 
of layering is called out in the requirements for B3 and A I 
systems when the NCSC evaluation criteria [7, p. 38] state 
that, "The TCD shall incorporate significant use of layering, 
abstraction and data hiding. Significant system engineering 
shall be directed toward minimizing the complexity of the 
TCD ..." 

The layered design of the VAX security kernel was based 
heavily on the Multics kernel design work of Janson [15] and 
Reed [28] and to a lesser extent on the Naval Postgraduate 
School kernel design |B]. Figure 3 shows a diagram of the 
layers of the VAX security kernel. The arrows in Ihc diagram 
indicate how each layer functionally depends on the abstract 
machine created by lower layers. 

Each layer adds specific functions within the security ker- 
ne), such that at the security perimeter, the secrecy and 
integrity models are enforced. The kernel itself is process 
structured, as described in the summary of the various lay- 
ers. Unlike many other kernels; all of the trusted processes 
run within the seenrily perimeter and are included in the 
format specifications described in Section S.4. 

HIH The Hard ware- Interrupt Handler layer is immediately 
above the physical VAX hardware and modified mi- 
crocode. It contains the interrupt handlers for the vari- 
ous I/O controllers and certain CPU -specific code. 
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Delete audit data 

Downgrade secrecy o* text alter human inspection 
Downgrade secrecy oi data without inspection 
Enable untrue ted kernel debugger 

Mount volumes, change printer paper, boot and shutdown system 
Register and change non-security attributes of devices, virtual 
machines, and users 

Control audit log and real-time alarms 

Enable or disable covert channel defenses 

Create, delete, or copy kernel files 

Change users' passwords and password parameters 

Upgrade integrity of text after human inspection 

Upgrade integrity of data eithout inspection 



Tabic 1: User Privileges 



Privilege 


Powers 


OPERATE 
SET -ACL 


Dismount volumes; activate and deactivate other virtual machines; set 
login limits 

Change any object's ACL, if access class permits 



Table 2: Virtual Machine Privilege* 



LLS The Lower- Level Scheduler is based strongly on Reed's 
two-lcvcl scheduler design (2fij. II creates Ihe abstrac- 
tions of level one virtual processors (vpls) that arc the 
basic unit of scheduling for ilie system. The LLS sup- 
ports symmetric multiprocessing by binding and un- 
binding real CI' Us io individual vpls. As shown in 
Figure 4, there arc th rem kinds of vpls: dedicated vpls 
that typically contain device drivers, bindablc vpls that 
can be bound to dedicated vp2s by the higher level 
scheduler, and addressable vpls that ran be hound to 
bindable vp2s ami thereby to virtual machines. Vpls 
are intended to be very inexpensive processes for usu 
within the kernel. Only addressable vpls have full ad- 
dress spaces; all oilier vpls run out of the global address 
spare of the kernel. Thus, the lowcr-levd scheduler can 
context switch in and out of most vpls by merely sav- 
ing registers and switching slack pointers. The lower - 
level scheduler also implements uvtsiitcounls |29j as the 
basic synchronization mechanism of tlic kernel. Event- 
counts can be. awaited or advanced in the normal way, 
or a processor inle.rru.pt can be tied to an cvciilcmmt, 
such that a V'M can be interrupted wli«:n an event count 
has reached a particular value. This processor interrupt 



mechanism provides upward transfers of control tliat are 
otherwise forbidden in the kernel. Processor interrupts 
arc only delivered when the CPU is executing uultudc 
the security kernel. 

JOS The I/O services layer implements device drivers that 
control the real I/O devices. The current version sup- 
ports only directly connected terminals and storage de- 
vices. 

VMP The VM physical memory layer manages real physi- 
cal memory, and assigns it to virtual machines. 

VMV The VM virtual memory layer implements the 
shadow page tables needed to support virtual mcinqry 
in the virtual machines. 7 VMV implements a primary- 
memory only strategy, requiring that all Hit: physical 
memory that a virtual machine sees be physically res- 
ident when that virtual machine is active. While this 
technique limits the number of simultaneously active 

'Shadow page Uhlns aft* created hy a VMM. because the pliyKtt-a! 
aridretww in page table entries must be relocated. Shadow paR«" La- 
blr* arc. described in detail by Madntck and Donovan |23, Section 9-S). 
Shadow page table? *nr also whcrr ring compression occurs. 
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Figure 3: VAX Security Kernel Layers 



virtual machines to the number that can lit into physi- 
cal memory simultaneously, it significantly ml mas ker- 
nel complexity by eliminating the need for a demand- 
paging mechanism in the kernel. It also eliminates the 
phenomenon of double paging that is often seen in other 
VMM*, where the demand paging mechanisms of the 
VMM and of the VMOS can thrash against one an- 
other, leading to serious performance degradation. In 
the VMM security kernel, the virtual machines arc al- 
located a fixed amount of physical memory and do all 
their own paging. 

HLS The Higher- Level Scheduler is also based on Reed's 
Iwolevcl scheduler |28|. Unlike Reed's design, our 
highcT-level scheduler is extremely simple because it 
docs not need to schedule access to primary memory. 
The HLS docs create the abstraction of level-two virtual 
processors (vp2s). There are two kinds of vp2s: dedi- 
cated vp2s that are used primarily by tint SSVR layer 
described below and bindable vp2s that are use*! for vir- 
tual machines. Figure 4 shows the relationships between 
vpls and vp2s. 

AUD The auditing layer provides the facilities for security 
auditing and security alarms. It is described in detail in 
a companiou paper |31|. 

FllF The Files II Files layer implements a subset of the 
ODS 2 file system that is also used in tho VMS op- 
crating system. 8 The most significant restrictions on 
the VAX security kernel implementation of ODS 2 arc 
that all files must be prc-allocalcd and contiguous. This 
reduces kernel complexity by eliminating the need for 
dynamic file extensions. FllF implements ODS 2 files 
only as a flat file system. 

VOL The Volumes layer implements VAX security kernel 
and exchangeable volumes and provides registries of all 
subjects and objects. These registries arc much simpler 
than ODS 2 directories. 

VTerm The Virtual Terminals layer implements virtual tnr- 
minals for each virtual machine, and manages the physi- 
caJ terminal lin«K. Each user may have multiple sessions 
connected to different virtual machines, and VTerm pro- 
vides the session management function**, as described 
in Section 4.1. VTerm also implements asynchronous 
network lines to allow virtual machines to connect to 
single- access- class networks via specially dedicated ter- 
minal lines. The current %*crsion of the system lias no 
support for higher- speed network connections. 

VPrint The Virtual Printers layer implements virtual print- 
ers for each virtual machine and multiplexes the real 
physical printers among the virtual printers, it provides 
top and bottom labeling, as well as trusted banner pages 
to delimit listings of different access classes and different 

VMs. 

* A brief summary nf tlm Files - 1 1 ODS 2 structure can be found in 

the appendices of |3S). 
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KI The Kernel Interface layer implements virtual controllers 
Tor the various virtu aj I/O devices and the security 
function controller, which implements such functions as 
loading virluaJ disks into virtual drive**. 

VVAX The Virtual VAX layer completes the virtualizaiion 
process by emulating sensitive instructions, delivering 
exceptions and interrupts to the virluaJ machine, etc. 

5SVR The Secure Server layer implements the trusted path 
for the security kernel, logs users in and out. and pro- 
vides security- related administrative functions. There 
is a dedicated vp2 for earJi tcrminaJ line to provide a 
Server process for each logged in user. 

VMOS The VMOS layer is the virtual machine's operating 
system. 

USERS The users in Figure 3 include both the unlrusted 
applications programs that run on top of the VMOS, 
and the human linings who communicate directly with 
the secure server via the trusted path. 




Figure 4: Level One and Lcvcl Two r Virtual Processors 



3.8 Software Engineering Issues 

A number of interesting software engineering issues arose 
during the development of the VAX security kernel. While 
space docs not permit dismissing all of them, this section 
highlights a few of the most significant. 

3.8.1 Programming Language Choice 

Perhaps the most critical software engineering issue in the 
VAX security kernel design was the choice of a programming 



language. From the. problems that KSOS li had with its 
choice of compilers |3, 25), it was clear thai wo needed high 
qualiLy compilers to develop our security kernel. While wc 
wanted as strongly- typed a language as possible, it was much 
more critical that the compiler correctly mm pile wy large 
programs, prodnnr high quality VAX object code, and be 
supported by an organization that could quickly respond to 
any problems wc might find. 

At the lime the VAX security kernel prototype effort be- 
gan, there were only a small number of sys terns progra ru- 
ining languages available for the VAX architecture: DLISS 
32, PL/I, PASCAL, and C. DLISS 32 was rejected because 
of its lack of data typing facilities. PASCAL was rejected 
because the V2.0 compiler that generated high fpiaJily code 
was not yet available. This left PL/1 and C, both of which 
used the same good quality code gc.utiraUir. Wc ch<»sc PL/ 1 
because of its slightly better data typing support, buuaitM! 
of its better support for character siring manipulation, and 
because the first prototype developers had -extensive prior 
experience in coding operating systems in PL/1. 

Wc were not happy with the choice of PL/I because its 
data types worn not strongly enforced. When the high qual- 
ity V2.0 PASCAL compiler became available, we began writ- 
ing new «:ode for the kernel in PASCAL. PASCAL provides 
much stronger data. type checking than PL/L and the VAX 
calling standard made inter- language calls easy to imple- 
ment. 

Higher-level language compilers cannot generate optimal 
code for all programs. Therefore, wc found it rio«:.:Nsary 
to implement those modules that actual me as u rem en Is had 
shown to be performance-critical in the MACRO 32 assem- 
bly language. Table 3 shows how much code was written in 
each of the languages for each layer of the kernel. 0 The tabic 
shows the number of executable source t:nde statements (ex- 
cluding comments, declarations, and white space) and pcr- 
layer and pcr-languagc totals. 

In retrospect, the use of both PL/J and PASCAL has led 
to certain difli cullies. Software engineers must be trninod 
in both languages, and some kernel bugs have resulted from 
misunderstandings of how to pass parameters from mm lan- 
guage to the other. Future security kernel developers would 
do well to choose one systems programming language and 
stick U» it. 

3.8.2 Coding Strategies 

A number of coding strategies proved very useful in the de- 
velopment of the VAX security kernel. For example., we 
avoided all use of global pools within the kernel to mini- 
mise the possibility of storage channels. The maximum si*c 
of data structures is determined at system boot time (based 

'T&hlc .1 indoors a numher of entries that are not jthowti in the 
layer diagram in Figure 3. TI.esr layers, COMMON, PMM, SVSBOO. 
VMMUOOT. ami VMM LID provide certain booting and runtime li- 
brary swipport functions. The normal runtime libra* ie* for the TL/1 
and PASCAL languages arc Dot linked into the kernel because they 
would have added a huge amount of onlc that Would need to be eval- 
uated and placed under configuration control. 
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Layer 


MACRO 


PASCAL 


PL/ 1 


Total 


WAX 


3371 


1502 


0 


4873 


SSVR 


0 


6876 


330 


7206 


KI 


10 


3354 


0 


3364 


VPRIHT 


0 


1455 


0 


1456 


VTEKM 


0 


1419 


0 


1419 


VOL 


0 


2553 


0 


2553 


Fl IF 


0 


2962 


0 


2962 


>UD 


0 


543 


0 


643 


HLS 


0 


0 


430 


430 


VNV 


129 


0 


3 069 


1198 


VHP 


0 


0 


352 


352 


XOS 


0 


4725 


0 


4725 


LLS 


1289 


13 


3839 


5141 


HIH 


815 


2393 


174 


3382 


COMMON 


244 


0 


0 


244 


PKH 


0 


0 


176 


176 


SVSROO 


2641 


734 


0 


3275 


VMMBODT 


55 


213 


430 


698 


VHJILIB 


3021 


503 


1265 


4789 


Total 


11475 


29245 


8066 


48786 



Tabic 3: Executable Statements per Layer 



on system generation parameters), and memory is allocated 
for that maximum size during kernel initialization. 

Different sections of memory within the kernel arc sep- 
arated by no-access guard pages to delect run-away array 
or string references. Unused memory is set to all ones to 
increase the chance of detecting the use of uninitialized vari- 
ables because zeros arc less likely to generate exceptions. 

The layers of the kernel are coded defensively with sanity 
checks to protect each layer from higher layers. If irregulari- 
ties arc delected, the system crashes to avoid the possibility 
of a security compromise. These sanity checks were devised 
to aid in the debugging of Ihe kernel and do not themselves 
provide security assurance mechanisms. However, many of 
the checks remain enabled in the finished kernel In help de- 
tect any remaining bugs. 

The actions of a user or a virtual machine cannot crash the 
kernel. They can cause error messages, exception conditions 
raised in the virtual machine, or in extreme cases, the halting 
of an offending subject. 

Since the entire TCD runs in kernel mode, there are 
no hardware-enforced firewalls between layers. However, 
the layering methodology forbids lower layers from calling 
higher layers. To help us spot layer violations, wc ap- 
plied both automatic and manual techniques. Using the fea- 
tures of the VAX DEC/Module Management System (VAX 
DEC/MMS) and the VAX DEC/ Code Management Systems 
(VAX DEC/CMS), we were able to isolate all dependencies 
of a layer on other layers. By visual inspection, we could 
immediately spot upward references. In fact during develop- 
ment, we did detect and fix several such occurrences. 



4 Human Interfaces 

High- security systems have developed a reputation for being 
hard to use, primarily due to their limited user interfaces. We 
believe thai il is essential that a human interface meet the 
expectations of today's commercial computer users. How- 
ever, wc faced the same obstacles faced by other developers 
of high- security systems: 

• Development resources are limited and satisfying the A3 
criteria takes precedence over all other efforts. 

• The kernel must he smaJI and verifiable. User interface 
features, such as a sophisticated command parser, arc 
large and often difficult to verify. Consequently, an in- 
terface built entirely on trusted code cannot match the 
usability of an interface built on unlruslcd code. 

We overcame these obstacles by creating two separate 
command sets: the Secure Server commands and the SE- 
CURE commands. The Secure Server commands arc imple- 
mented entirely in trusted code. The administrative com- 
mands, the SECURE commands, are parsed in the VMS 
and ULTR1X 32 operating systems. With this approach, 
we reduce the amount of trusted code and gain the well- 
developed command interfaces of these mature commercial 
operating systems. SECURE commands arc normally only 
issued by the system manager, the security manager, the op- 
erators, and the auditors, although ordinary users may need 
to issue a few of them at times. Dy contrast, all users must 
issue some Secure Server commands to login and connect to 
virtual machines. 

4.1 Secure Server Commands 

The Secure Server is the user's direct interface lo the kernel. 
A user invokes a trusted path to the Secure Server by pressing 
lhc5<;rure Attention Kny. This key operates at all times and 
cannot be intercepted by nn trusted code. We have chosen 
the BREAK key lo be the Secure Attention Key. 

The Secure Server's commands control terminal connec- 
tions to virtual machines in the same way that a terminal 
server controls terminal connections to physical machines, 
using commands such as: CONNECT, DISCOHNECT, RESUME, 
and SHOW SESSIONS. A user can create sessions with several 
virtual machines at different access classes and can quickly 
switch from one to another. 

The interface for the Secure Server commands is built en- 
tirely in trusted code and offers only minimal command-line 
editing functions. 

4.2 SECURE Commands 

The tools for managing the system arc the SECURE com- 
mands. The SECURE commands and utilities arc im- 
plemented just as arc other commands in the VMS and 
ULTRIX- 32 command languages, except that they issue ker- 
nel calls to do their work. The complete set of SECURE 
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commands and utilities is installed i» the VMS operating 
system. A subset of the SECURE commands is offered by 
the ULTIUX-32 operating system. 

The SECURE commands, nnlikr tin: Secure Server com- 
mands, arc parsed by the VMS and ULTRJX 32 command 
language interpreters. The user can take advantage of such 
features as command-line recall and command procedures. 

There arc two types of SECURE commands: VM 
SECURE commands and Uskt .SECURE commands. Doth 
types of SECURE commands arc issued from the VM's 
operating-system commaud level. VM SECURE commands 
arc executed in the context of the issuing VM. User SECURE 
commands are submitted to the Secure Server for execution. 
The commands arc distinguished by the type of subject, a 
user or a virtual machine, that holds the access class and 
privileges necessary to issue, the command. 

4.3 Command Confirmation 

While both the User and VM SECURE commands arc ad- 
ministrativc commands, only the User SECURE commands 
must Ik-, trusted. For such security- relevant commands, we 
require A 1 assurance that: 

• The ro • ii m and was issued by a user and not by a Trojan 
horse in a VM. 

• The command received by the Secure Server is exactly 
the same command typed by the user and not a com- 
mand that was covertly modified by a Trojan horse. 

• The user who issued the command can be identified in 
the audit log. 

Our design for the User SECURE commands provides 
both trust and individuality accountability even for com- 
mands issued from an nn trusted environment. Upon receipt 
of a valid User SECURE command, the VM instructs the 
user to press SECURE ATTENTION. This key invokes a 
trusted path between the user's terminal and the Secure 
Server. A SECURE ATTENTION signal can be sent to the 
Secure Server only by manually pressing the DREAK key. 
This prevents a Trojan horse from completing the cxec.nl km 
of a User SECURE command. 

To prevent a VM from spiffing the user by passing a dif 
fcrcnt cmuiitaiid from what the user typed, the Secure Server 
displays the action that will be taken by the command and 
prompts the user to approve or reject the operation. Figure 5 
is an abbreviated example of a User SECURE command is- 
sued from a VMS virtual machine. Resuming indicates that 
control of the terminal will be returned to the virtual ma- 
chine. 

4.4 SECURE Utilities 

Managing the VMM security kernel requires a number of 
utitilics. Our SECURE utilities are modeled after VMS util- 
ities and arc summarized in Table 4. 



$ SECURE DELETE TLS : STATUS . RPT 
PrwBs SECURE ATTENTION to complete 

execution of this command. 
User presses SECURE ATTENTION to establish, a 

trusted palh. 
Delete VAX security kernel file 

TLS: STATUS. RPT 
Confirmation lYes or No) : Y 
VMM: File deleted 
Resuming. . . 

Figure 5: Example of a User SECURE command 



SECURE Utility 


Purpose 


Authorize 

Register/ Device 
Rcgi s t ct / Vol ume 

Sysgcn 

Crash Dump Analyzer 


Rirgislcrs users and virtual 
machines, etc. 
Registers I/O devices. 
Registers disk and tape 

volumes. 

Sets limits on system resources. 
Provides data for determining 
the cause of a system crash. 



Table 4: SECURE Utilitic 



4.5 Reclassifying Information 

Users can be permitted to change the access class of the 
contents of a VAX security kernel file or an exchangeable 
volume with the SECURE RECLASSIFY command. This 
command copies the contents of a kernel file or volume to an 
existing kernel file or volume labeled with a different access 
class. The source and destination objects must lie within the 
user's access class range. In addition, privileges arc inquired 
if the reclassification downgrades the data's secrecy class or 
upgrades its integrity class. 

Reclassification normally requires trusted inspection by 
the user. Inspection is required to be sure that a Trojan 
horse has not inserted additional information that the user 
did not intend to reclassify. To make inspection easier, the 
user can opt to print the VAX security kernel file or display 
the file on the terminal, one screen at a lime. Once the 
complete file is printed or displayed, the user is prompted 
to approve the reclassification. To prevent the covert pass- 
ing of information from the source file to the target file in 
the form of invisible escape sequences, inspected files mnst 
contain only printing characters, spaces, and form feeds. A 
line may not end with a space because a trailing space would 
be invisible. The reclassification is terminated if any illegal 
character is encountered. 
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5 Assurance 

The principal reason for building an Al security kernel is to 
provide a high degree of assurance that the security features 
of the system actually work correctly. This section describes 
Borne of the techniques that we have used in the VAX se- 
curity kernel to provide the necessary assurance of security, 
to meet both the requirements of an A 1 evaluation and the 
requirements of real-world users. It is this integration of 
both Al requirements and real- world requirements that is 
of particular research interest, as previous security kernels 
have not succeeded at integrating the Al requirements with 
good performance and compatibility with large amounts of 
existing commercial software. 

Gasscr (10, p. 163] describes Honeywell's STOP kernel for 
the SCOMP [9) and Gemini Computers' GEMSOS |32) as 
commercial-grade security kernels. However, STOP does 
not provide software compatibility with existing operating 
systems, and GEMSOS to dale has only been used in spe- 
cialised environments. Shocklcy, Tao, and Thompson [32| 
report that research is under way to provide both UNIX 
and MS-DOS environments for GEMSOS, but it is not dear 
whether those environments are yet working. If Gemini suc- 
ceeds in providing both UNIX and MS-DOS environments 
in GEMSOS, they will have succeeded al integrating A 1 re- 
quirements with real- world requirements. The VAX security 
kernel supports both the VMS and ULTIUX- 32 operating 
systems with their layered applications today. 



5.1 Design and Code Changes 

Every change to our code undergoes both design and code re- 
view, regardless of whether the code is trusted or untrusted, 
or whether it is a whole new layer or a bug fix. Design 
reviews for even the smallest fixes ensure that syslcm-wide 
effects are considered. Each layer has an owner, who partici- 
pates in the design review, and is responsible for the quality 
of that layer. Each code change is reviewed both in the con- 
text or its own layer and in the contexts of its calling and 
called layers, so as to catch inter-layer problems. 

Reviewers learn from the code they review, as well as shar- 
ing their knowledge through review comments. Reviewers 
address readability and clarily, security, performance, ele- 
gance and adherence to guidelines. Much like access con- 
trols, design and code guidelines arc either mandatory or 
discretionary. Mandatory guidelines arc based on prior expe- 
rience in security kernel developments. Discretionary guide- 
lines are used to avoid well-known traps in the programming 
language, and to produce consistent, readable code. This 
consistency makes it easier for an engineer to pick up and 
debug in a new area, reducing engineering costs and time. 

The code review results, along with the design and lest 
plan, are publicised for the entire group to check. Tliis prac- 
tice provides a last review of the entire change by a large 
audience. Code review results can also serve as examples 
from which engineers can learn good coding practices. 



The development team makes extensive use of VAX Notes 
online conferences to publicize design and coding guidelines, 
to discuss specific design issues, to track bug reports, and 
to record and publicise the results of the above-mentioned 
design and code reviews. 

Each coding task is integrated with the current working 
system as soon as it is complete. This integration always 
produces a working system. (See Section 5.3.) Continual and 
incremental integration avoids major unexpected failures by 
identifying design and/or coding errors as soon as possible. 

5.2 Development Environment 

As mentioned in Section 2, we have been developing the VAX 
security kernel on a VAX security kernel system. Thus, our 
group does its daily work on a system designed to meet Al 
security requirements, using most of its features and con- 
trols. Our VMs run at meaningful access classes. Different 
versions of the kernel are maintained on different VMs to 
keep orthogonal tasks from impinging on each other. We also 
use VMs for developing and testing the untrusted code that 
must run in the VMS and ULTRIX-32 operating systems. 
We have separated the roles of our own system manager and 
security manager, as recommended in the NCSC Evaluation 
Criteria |7|. 

The CPU and console of the development machine arc 
Kept inside a lab that only members of the VAX security 
kernel development group can enter. Within that lab, the 
development machine is protected by a cage, which consists 
of another room with a locked door. Physical access to both 
the lab and to the cage within the lab is controlled by a 
key-card security system. Finally, our development machine 
is not yet connected to Digital's internal computer network, 
to minimize the external threat to our development environ- 
ment and our project. 

5.3 Testing 

Integrating a coding task requires that a developer run a 
standard regression test suite. Integration occurs usually at 
least once a week, and as often as twice a day. 1 " This regres- 
sion suite consists of two portions: layer tests and KCALL 
tests. Layer tests are linked directly into the kernel, and 
test layer interfaces and internal routines by calling them di- 
rectly and checking their outcome. KCALL tests run in a 
VM, issuing legal, illegal, and malformed requests, to check 
the VM interface. 

A separate suite of tests, issued via the VAX DEC/Test 
Manager (DTM), is run once every two weeks to test the user 
command interface. These tests currently run for 30 hours. 
They consist of commands thai arc successful, commands 
that produce errors, and commands that send malformed 
packets to the SSVR layer. DTM checks both the results of 
each command and the displays it produces. 

We also run the standard VAX architecture exerciser 
(AXE) that verifies that a particular CPU correctly implc- 

,0 Dcvclopcra of course run individual tests prior to integration. 
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ments a VAX computer. Wc run AXE to lest the VAX 
virtualissalion, described in Section 3.2. AXE tests were run 
extensively during the development of Hie CPU microcode 
extensions and the WAX layer. They will be run again 
when the kernel reaches final cum plot itm. 

We are currently developing test plans for fully exercising 
ali of the access control derisions and other security -relevant 
checks made by the system and for system-penetration lest- 
ing. Some of these now tests will be developed from scratch, 
and some will be based on the formal specifications. 

5.4 Formal Methods 

The requirements for an A I security evaluation itate that a 
formal security policy model must be written, that a formal 
top-level specification (FTLS) of the system design must be 
written and proven to satisfy the security policy model, that 
the system implementation must be informally shown to be 
consistent with the FTLS, and thai formal methods must 
be used in covert channel analysis of the system. The FTLS 
must accurately model system external interfaces, externally 
visible behavior, and security- relevant actions. A descriptive 
top-level specification (DTLS) is also required as a complete 
natural language description of the system. 

We use the Formal Development Methodology ^FDM.) 
specification and verification system (19] to help meet these 
requirements. Wc arc writing both our security policy model 
(which consists of criteria and constraints and the lop- level 
specification (TLS) of the various transforms) and our FTLS 
in the FDM specification language, Ina Jo. Wc are using 
the FDM interactive theorem provcr (ITP) to show that the 
TUS obeys the policy and that the FTLS maps to the TLS. 
The DTLS consists of our internal design diM-.n mentation, 
plus some special glue documents thai tic the DTLS and the 
FTLS together, particularly describing areas of the kernel 
that arc not formally modeled in the FTLS. 

Table H shown the. number of executable statements in the 
security kernel. For comparison, table S shows an estimate 
of the total number of lines of lna Jo (comments excluded) 
and the number of lines of transforms (declarations excluded) 
required to specify that kernel. The numbers are estimates 
because the FTLS is not yet complete. The totals show that 
the number of lines of transforms arc about one sixth of the 
number of executable statements in the security kernel. 



Level of Specification 


Lines of lna Jo 


Total 


Transforms 


TLS 
FTLS 


6S0 
11758 


294 
8410 


Total 


12408 


8704 



Table 5: Lines of Formal Specifications 

Wc arc doing a formal covert channel analysis using a new 
technique for automating the Shared- Resource Matrix ap- 
proach [20] using code-level flow analysis tools. 



Formal methods do not make the system secure by them- 
selves. Successful proof that our specifications meet security 
policy does not guarantee that there are no lurking imple- 
mentation bugs. However, the use of formal methods sig- 
nificantly improves the overall quality of the security ker- 
nel. When combined with the informal testing pnvcctdiinrs of 
Section 5.3. the use of formal methods improves the assur- 
ance thai the security features arc effective. Indeed, the very 
act of formally specifying the security kernel in Ina Jo has 
already detected several kernel bugs, both because of con- 
straints imposed by proof procedures, and because the. pro- 
cess of code correspondence provides a thorough method for 
reviewing the: TCR code and informal design specifications. 
The separation of duties between the software engineer and 
the verifier, by itself, provides valuable extra assurance, even 
if no proofs were ever done. 

5.5 Configuration Control 

Wc maintain strict configuration control on many items, in- 
cluding design documents, trusted kernel code, test suites, 
user documents, and verification documents. All of our code 
is maintained under the VAX DEC/Code Management Sys- 
tem (CMS) to maintain a history of cadi change to each 
module. Security reviews check each item against the specific 
NCSC criteria requirements |7| it fulfills and check among 
the items for internal consistency. Horns that have boon re- 
viewed arc stored on a master pack that is physically pro- 
tected against modification. 

Our hardware, firmware, and software development tools 
are developed by other groups within the corporation. We 
review hardware and firmware ECOs, prior to supporting 
them in the VAX security kernel. New versions of software 
development tools arc tested on a sUtid- alone laboratory sys- 
tem prior to use on the kernel development machine. Wc use 
only lite standard, released versions of software development 
tools, the same versions that have been checked out for ship- 
ment to our customers. With rare exceptions, no field-test 
versions arc permitted on the kernel development machine. 

5.6 Trusted Distribution 

The end user of a security kernel must have some assurance 
that no one has tampered with or substituted counterfoil 
copies of the hardware and software that make up the system. 
Hardware and software have different trusted distribution 
requirements. 

5.6.1 Hardware Trusted Distribution 

To assure that the hardware systems would arrive at the 
customer's site meeting the trusted distribution criteria, we 
have developed a security-seal program. If someone tam- 
pered with the seal, evidence would be provided of the at- 
tempted entry. A locking device would combine with the se- 
curity sealing procedures to ensure a trusted shipment. Full 
individual accountability would be provided, including logs 
of the delivery. 
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5.6.2 Software Trusted Distribution 

Installation of an A I system involves achieving a trusted 
state. The steps lo do this on VAX 8800 hardware are com- 
plex. The console processor software and CPU microcode 
must be installed and cryptographic ally checksummed with 
stand-alone software to detect any possible tampering. If a 
secure site loses its trusted state for any reason, they must re- 
install the console software and theCFU microcode. Trusted 
state could be lost just by running an un trusted operating 
system or hardware diagnostics on the system. 

Next, the trusted code is installed via untrusted code 
(VMS) and the result is cryplographically checksummed to 
verify that the untrusted code has not tampered with the 
trusted code. The result of the checksum is checked against 
a message authentication code to verify correct installation. 
The checksumming software is shipped separately from the 
rest of the software, so that a single failure of the trusted dis- 
tribution system could not compromise both the checksum 
program and the authentication code. 

For software, there would also be an option of using trusted 
couriers instead of the separate delivery paths. 

6 Production- Quality Kernels 

A production-quality security kernel is designed to protect 
and ensure the quality of real-world information. This sec- 
tion describes some of the differences between research and 
production-quality security kernels that are required to meet 
general user requirements, as well as to satisfy the NCSC cri- 
teria for an Al operating system. 

6.1 Producing the Kernel 

The primary tools for creating a security kernel are compil- 
ers. Quality compilers must work for large programs, pro- 
duce efficient object code, and be reliably supported. We 
sacrificed programming language elegance in favor of com- 
pilers with a strong track record: the VAX PASCAL and 
PL/I compilers. We maintained contact with the compiler 
developers throughout the development, and they provided 
much needed help to us, including occasional changes to the 
actual compiler code. 

A second tool, a symbolic debugger/crash dump analyzer, 
is needed to develop and debug the system. It would also be 
needed by users and support personnel to diagnose problems, 
and by users who might wish lo add functions to the kernel. 

A production -quality security kernel must have adequate 
performance to justify its purchase in the face of other op- 
tions such as multiple separate computers or periods pro- 
cessing. To help ensure attention Lo performance, we do our 
own development work on a VAX security kernel system. 
Performance-critical paths were written in a high-level lan- 
guage and then re-written in assembly language for speed. 
We have meters to find performance- critical routines, and 
a rudimentary performance monitor to gather statistics on 
CPU and I/O usage. 



Dug tracking mechanisms arc needed both to satisfy NCSC 
configuration management guidelines, and to give us a incan6 
to respond to problems on a timely basis. They also provide 
a means to check against our definition of quality: having no 
security bug* and no bug that keeps production work from 
running. Statistics on the number of bugs and their severity 
provide concrete feedback on stability. 

6.2 Documentation 

A real security kernel requires extensive doc u mentation for 
its users and for its system and security managers. These 
documents must not only meet the content requirements of 
(he NCSC; tlicy must also be clear and understandable to 
both novice and sophisticated customers. The VAX security 
kernel documentation set consists of nine manuals and a ref- 
erence card. The manuals include a user's guide, guides to 
both system security and system management, a command 
reference manual, both basic and advanced programmer's 
manuals, an installation guide, a master index, and release 
notes. Thnse manuals have been written to the same qual- 
ity standards as the manuals for the VMS and ULTRIX 32 
operating systems. 

7 Comparison with KVM/370 

- Wliilc the VAX security kernel superficially bears a strong 
resemblance to KVM/370, in that both systems create vir- 
tual' machines that run at different access classes, the internal 
structures of the two systems arc very different. 

Most significantly, KVM/370 was designed as a retrofit 
to the existing VM/370 product, with a specific goal of 
leaving at least half of the original code intact (ll|. As 
a result, KVM/370 was structured as shown in Figure 6. 
The KVM/370 security kernel used a variation on self- 
visualization to create a scries of NKCPs (Non-Kcrncl Con- 
trol Programs), each at a distinct mandatory access class. 
The NKCPs ran unmodified VM/370 code to create multi- 
ple virtual machines that then ran the CMS (Conversational 
Monitor System), a single- user operating system designed 
tu run in a virtual machine. The disadvantage of this ap- 
proach is that many functions executed by a virtual ma- 
chine required two context switches, first into the NKCP 
and then into the security kernel. By comparison, VAX se- 
curity kernel achieves a higher performance level by allowing 
the virtual machines to communicate directly with the se- 
curity kernel. This makes the VAX security kernel larger 
than the KVM/370 security kcrnrJ, lint we believe that the 
performance gains jnslify the increase in size. 11 

KVM/370 never implemented support for VMOSs that 
supported virtual memory. It implemented demand paging 
within its TCD. Dy contrast, the VAX security kernel leaves 
virtual memory support lo the VMOSs. As discussed in 

''This comparison is not strictly fair to KVM/370 because the 
KVM/370 team was required to maintain compatibility and a large 
body of origin Hi code from VM/370, while the VAX security kernel 
team bad the liberty of designing and coding from scratch. 
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Section 3.7, eliminating demand paging reduces kernel com- 
plexity and improves performance at the cos! of limiting the 
number of simultaneously active virtual machines. 

Another major difference is that KVM/370 lias a very lim- 
ited interface for system management and security manage- 
ment functions. For example, new user* cannot be added 
during online operation. By contrast, the VAX security ker- 
nel offers a full complement of system and semrity manage- 
ment tools, such as arc required in a gen craJ- purpose system. 
(See Section 4.) 

While performance comparisons are very trirJcy to make, 
the relative performance of the VAX security kernel seems 
better than that of KVM/370. KVM/370 reports |ll] per- 
formance ranges fxoin 10% to b0% of VM/370, depending 
on the workload. By contrast, the VAX security kernel ex- 
hibits jM-rformancc ranges from 30% to 00% of VMS capacity, 
again depending on the workload. The KVM/370 measure- 
ments were of an untuned system, while the VAX security 
kernel measurements were of a system with a I i mitral amount 
of tuning. Tim KVM/370 comparisons were to VM/370, it- 
self a virtual- machine monitor with performance degradation 



compared to a nalivc operating FyEieni. The VAX security 
kernel comparisons were to the native VMS operating S3's- 
trrn. KVM/370 reported * number of desirable performance 
optimizations that had not lusrn done, and likewise, we know 
of a number of optimizations that have not yet been applied 
to VAX security kerne! because of limited development re- 
source*. 

8 History of the Project 

The idea of a virtual-machine monitor security kernel for the 
VAX, similar in concept to KVM/370, was first conceived 
by Paul Kargcr and Steve Lipncr in a Mexican restaurant in 
Palo A Uo, CA, i mined lately after the J 981 Symposium on 
Security and Privacy. An initial design study (17) concluded 
in 1982 that such a security kernel would be practical for the 
VAX architecture. 

The security kernel was initially prototyped on a VAX- 
1 1/730 system. The VAX ] 1/730 CPU (34) was particularly 
attractive because it was vertically microprogrammed, and 
its microcode was executed from a writcabic control store 
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(WCS) that could be reloaded from magnetic tape cassettes. 
This environmcnl was ideal for experimenting with aJlcrnatc 
microcode extensions to the VAX architecture, although the 
CPU itself was quite alow. 

The VMS operating system first successfully booted in a 
virtual machine on 39 July 1984. That version of the security 
kernel was a research prototype and was not a production- 
quality system. It was extremely slow (due in part to the 
choice of the VAX-11/730 and in part to the initial soft- 
ware design that cmphaais&ed quick development and ex ten- 
sive self-checking, but not performance), and its user inter- 
face was extremely crude. 

Once the VMM security kernel prototype was running re- 
liably on the VAX-11/730 and we had accomplished some 
performance tuning (that improved system performance by 
at least an order of magnitude), we then began investiga- 
tion of what a production-quality version would be like. The 
extensions to the VAX architecture were re-implemented on 
the VAX 8800 family of CPUs to provide a liigh-performance 
base for the system. Like the VAX-11/730, the VAX 8800 
CPU (24] runs its microcode from a writeablc control store 
(WCS), so modifications were possible. The VAX 8800 mi- 
crocode is organized horizontally, rather than vertically, and 
the micrijcode is pipelined, bo the actual implementation of 
the extensions was much more complex than for the VAX- 
1 1 /730. 

Going from the research prototype to the practical version 
also gave us the opportunity to revisit a number of design 
decisions. In particular, the extensions to the VAX arch- 
itecture to support visualisation were simplified, in part 
due to the limited availability of microcode memory in the 
VAX 8800. A performance study of the VAX security ker- 
nel prototype revealed that some of our architectural exten- 
sions did not provide the expected performance gains, while 
other extensions would be more valuable. For example, the 
prototype design included complex microcode assistance for 
delivering exceptions and interrupts to the virtual machines, 
but these microcode assists proved not to be useful, and a 
much simpler Bcbeuie was implemented for the VAX 8800. 
Siin ; !arly, performance measurements of the prototype re- 
vealed that VAX operating systems (and VMS in particular) 
use the MTPR instruction to change their interrupt priority 
level (IPL) much more frequently than anyone had expected. 
Therefore, the software was changed to optimize this particu- 
lar path, and microcode assistance was considered, although 
not implemented in this version. 

The move to the product] on- quality kernel also marked 
the development of such features as user and system man- 
agement interfaces, auditing, and error logging. The proto- 
type kernel, as a research kernel, had no need of such tools, 
hut a real Al system must have them, so that the end users 
can manage and reliably run real applications on the system. 

Dy January 1988, the kernel was sufficiently stable that 
some engineers could begin doing their development work on 
a VM. Also in January 1988, the first VAX security kernel 
was installed outside the kernel development gronp. That 
system was installed in the European ULTIUX Engineer- 



ing Group in Reading, England for porting the ULTRIX-32 
operating system to a virtual machine. ULTRIX 32 first 
booted in a virtual machine on 15 February 1988, only two 
months after detailed design for the port began, and less 
than one month after a working VAX security kernel system 
was available for use in Reading. 

Dy August 1988, VAX security kernel builds were being 
done on virtual machines, and by early 1989, essentially all 
software development work was being done on the kernel. 
By Spring of 1 989, the kernel was sufficiently stable that 
the VAX 8800 that had been running a conventional VMS 
time sharing system for the kernel developers was released 
for other purposes. 



9 Conclusions 

The VAX security kernel is a working, production-quality 
VMM security kernel with performance sufficient to support 
a large number of time-sharing users. It is sufficiently fast 
and stable that it supports its own development team. II 
supports vast amounts of existing user software that has been 
written for both the VMS and the ULTRIX-32 operating 
systems, and it supports both operating systems running 
simultaneously on the same CPU. VAX security kernel is 
currently (as of February 1990) in the Design Analysis Phase 
with the National Computer Security Center (NCSC) for an 
Al rating. As a research project in what is required to build 
a practical seenrity kernel, it has been a major success. 

The development of VAX security kernel has been long 
and arduous, and we have learned a number of lessons dur- 
ing that time. Performance of a security kerne) is extremely 
important, and getting good performance is very hard. It 
requires detailed analysis of what portions of the kernel arc 
performance-critical and a willingness to redesign those por- 
tions for performance and possibly re-code them in assembly 
language or to provide microcode performance assistance. 

Building the system twice, once as a research prototype 
and once as a research study of a production-quality system, 
was extremely valuable. The second time around, we were 
able to apply some of the performance lessons learned by 
adjusting our microcode assistance, and we developed the 
user and management interfaces that are essential in a real 
system. 

Developing a system to Al standards is very hard work. 
Some of the Al requirements can directly conflict with per- 
formance and usability goals, and the testing and review 
requirements arc very time consuming. Furthermore, the 
export controls imposed on Al systems can seriously reduce 
the potential market for a system, making it difficult to re- 
cover the costs in achieving the A3 rating. On the other 
hand, the discipline required to meet A3 requirements defi- 
nitely improves overall software quality and reliability. 
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